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ABSTRACT 

We  present  a  summary  of  the  drawbacks  and  deficiencies  that  we  noticed  in  the  currently  available 
Command  Support  Systems  (CSS)  and  the  methodology  we  propose  to  improve  them:  Situation  Awareness 
support  through  Cognitive  Fusion  of  Information  stemmed  out  of  document  analysis.  Our  approach  is 
divided  into  two  parts:  a  methodology  for  situation  representation  out  of  document  analysis  and  a 
methodology  for  situation  analysis  and  reasoning  to  support  decision-making.  The  situation 
representation  part  is  based  on  the  use  of  conceptual  graphs  and  fusion  of  nodes  in  graph  structures, 
whereas  the  situation  analysis  part  follows  Complex  Event  Processing  methodology. 


1.0  INTRODUCTION 

In  current  information  systems,  a  large  amount  of  data  is  available  for  giving  clues  about  a  situation  and  its 
evolution.  Furthermore,  as  these  data  stem  from  different  sources,  the  vision  they  give  is  more  global  and 
is  made  up  of  several  points  of  view  of  the  situation.  Being  aware  of  all  these  information  sources  is  a  big 
benefit  for  the  operator  in  charge  of  making  decisions  because  the  situation  is  then  more  fully  described. 
In  the  meantime,  it  is  also  a  major  drawback  because  such  an  amount  of  data  to  analyse  induces  cognitive 
overload  and  thus  errors  in  perception,  comprehension  and  projection. 

Good  decision  necessitates  a  clear  vision  of  the  ongoing  situation,  so  an  efficient  Command  Support 
System  should  provide  it  as  well  as  cognitive  function  to  support  decision-making. 

In  this  paper,  we  present  a  summary  of  the  drawbacks  and  deficiencies  that  we  noticed  in  the  currently 
available  Command  Support  Systems  (CSS)  and  the  approach  we  propose  to  improve  them.  The  first 
section  presents  the  notion  of  information  superiority,  as  well  as  the  limitations  of  the  state  of  the  art  CSS 
and  our  vision  to  solve  them. 

In  the  second  section,  we  give  a  general  presentation  of  our  approach  to  Cognitive  Information  Fusion  for 
Command  Support  Systems1.  This  approach  is  divided  into  two  parts:  a  methodology  for  situation 
representation  out  of  document  analysis,  which  is  described  in  more  details  in  the  third  section,  and  a 
methodology  for  situation  analysis  and  reasoning  to  support  decision-making,  described  in  the  fourth 
section.  We  then  conclude  and  briefly  introduce  our  future  works. 


1  Some  elements  of  the  theoretical  work  presented  here  are  partly  funded  by  the  ICIS  research  project  under  the  Dutch  BSIK 
Program  (BSIK  03024). 
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2.0  COMMAND  SUPPORT  SYSTEMS:  STATE  OF  THE  ART 

2.1  Information  Superiority:  A  new  challenge  for  Command  Support  Systems 

The  aim  of  a  Command  Support  System  (CSS)  is  to  support  decision-making  by  providing  the 
Commander  with  an  edge  over  knowledge.  To  reach  that  objective,  one  of  the  main  challenges  is  to  make 
the  operator  and  CSS  pair  more  efficient  at  the  lowest  possible  cost  by  providing  core  Information 
Superiority2  functions  such  as  Situation  Awareness  (SAW),  Common  Relevant  Operative  Picture  (CROP) 
or  Common  Operational  Picture  (COP).  We  claim  that  one  of  the  key  issues  is  to  build  a  good  situation 
understanding  based  on  a  context  sensitive  information  interpretation  and  through  a  well-constructed 
situation  representation. 

The  understanding  of  the  situation  gains  from  the  total  sum  of  relevant  information  provided  to  make  a 
correct  decision-making  regarding  the  allocated  objectives  and/or  the  desired  state.  Further  more,  a  large 
amount  of  different  information  sources  such  as  textual  or  radio  messages,  raw  data  radar,  images  and 
optical  images  are  now  available  in  order  to  give  clues  about  an  ongoing  situation. 

The  result  of  this  variety  of  information  sources  causes  information  inundation.  The  deficiency  of,  or 
errors  in,  situation  understanding  are  important  reducing  factors  to  information  superiority,  potentially 
provoking  decision-making  errors  which  could  have  been  avoided  if  operators  had  correctly  analysed  the 
situation  in  time.  Such  errors  occur  in  the  following  levels:  [1][2] 

•  Perception:  A  person  may  wrongly  perceive  task-relevant  information  or  not  perceive  it  at  all. 
Contributing  factors  are  related  signal  characteristics,  perception  strategies  in  complex 
environments  or  faulty  expectations. 

•  Comprehension:  A  person  may  wrongly  interpret  perceived  information.  Contributing  factors  are 
misuse  or  non-existence  of  proper  mental  models  of  the  environment. 

•  Projection:  A  person  may  wrongly  predict  a  future  status,  in  the  case  of  a  good  mental  model  but 
memory  limitations  for  instance. 

Errors  may  also  occur  due  to  information  overload.  For  instance,  an  operator  in  a  stressful  context  may 
focus  on  a  piece  of  non-significant  or  incomplete  information  and  not  on  the  relevant  information. 

Therefore,  given  an  objective  and  to  obtain  a  well  constructed  situation  representation,  one  needs,  at  first, 
to  extract  relevant  information  items  among  the  large  amount  of  incoming  information.  These  items  have 
then  to  be  grouped  in  order  to  obtain  a  synthetic  view  of  the  ongoing  situation. 

The  challenge  of  an  automated  construction  of  relevant  SAW  is  to  enhance  operators'  capabilities  in 
achieving  information  superiority  needed  to  dominate  modem  warfare.  Meeting  this  challenge  would 
allow  CSS  to  be  more  efficient  through  the  reduction  of  the  time  needed  to  construct  a  good  situation 
awareness,  as  well  as  a  reduction  of  information  overflow  to  the  operator.  To  do  so,  we  have  to  construct  a 
good  model  for  representing  the  operator's  mental  analysis  process  in  order  to  reach  the  quality 
requirements  for  this  automatic  construction  of  the  situation  representation,  including  heterogeneous 
information  fusion. 


2  The  concept  of  Information  Superiority  is  defined  by  the  CJCS  (Chairman,  Joint  Chiefs  of  Staff)  in  Joint  Vision  2020  (U.S. 
DoD)  as  “the  capability  to  collect,  process,  and  disseminate  an  uninterrupted  flow  of  information  while  exploiting  or  denying 
an  adversary’s  ability  to  do  the  same.” 
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2.2  Limitation  of  Current  Command  Support  Systems 

More  and  more  information  is  spread  across  a  variety  of  channels,  in  a  range  of  structured  formats  (signal, 
image,  database,  GIS)  and  unstructured  formats  (text,  speech,  video).  But  often  the  vital  pieces  of 
information  are  also  spread  across  several  formats,  and  must  be  combined  in  accordance  with  (i)  the  final 
goal  of  situation  awareness,  (ii)  their  reliability  and  relevance,  (iii)  the  semantics  of  these  formatted  pieces 
of  information,  in  order  to  get  the  complete  set  of  information  desired  at  the  right  time.  Up  to  now,  each 
format  has  been  processed  on  the  basis  of  appropriate  and  distinct  theories.  Moreover,  there  is  a  deficiency 
of  a  sound  theoretical  but  computational  framework  for  modelling  the  understanding  of  the  situation  and 
for  integrating  disparate  relevant  sources  of  information  in  order  to  support  the  ongoing  phenomena 
analysis  and  to  avoid  cognitive  overload. 

No  model  is  available  to  support  an  in-depth  understanding  of  the  situation  in  order  to  assist  with  decision¬ 
making  in  complex  situations.  Furthermore,  due  to  limitations  of  information  coming  from  a  single  source 
(uncertainty,  missing  observations,  etc),  there  is  a  lot  to  gain  by  integrating  and  fusing  multi-source 
information. 

2.2.1  Deficiency  of  semiotic  cognitive  information  processing  for  situation  understanding 

A  good  representation  of  the  situation  needs  to  capture  a  number  of  aspects  of  the  situation:  the  essential 
entities,  their  organizational  and  spatial  structures,  and  the  possible  actions  that  can  be  taken  to  affect 
internal  states  and  external  environments.  "Gestalt  principles"  and  "Situation  Theory"  model  individuals, 
properties,  relations,  spatio-temporal  locations,  and  situations. 

Gestalt  theory  [3]  arose  in  1890  as  a  reaction  to  the  prevalent  psychological  theory  of  the  time  -  atomism. 
Atomism  examined  parts  of  things  with  the  idea  that  these  parts  could  then  be  put  back  together  to  make 
wholes.  Unlike  atomists,  Gestalt  theorists  were  intrigued  by  the  way  our  mind  perceives  wholes  out  of 
incomplete  elements.  The  main  vision  of  the  Gestalt  theory  is  that  things  are  "more  than  the  sum  of  their 
parts"  [4].  Gestaltists  believed  that  context  is  very  important  in  perception  and  therefore  in  understanding  a 
situation.  Gestalt  theory  is  widely  studied  in  the  Fluman-computer  interaction  community. 

Situation  theory  is  a  mathematical  theory  of  meaning  [5].  According  to  the  theory,  individuals,  properties, 
relations,  spatio-temporal  locations,  and  situations  are  the  basic  ingredients.  The  world  is  viewed  as  a 
collection  of  objects,  sets  of  objects,  properties,  and  relations.  Individuals  are  conceived  as  invariants; 
having  properties  and  standing  in  relations,  they  persist  in  time  and  space.  Objects  are  the  parts  of 
individuals.  (Words  are  also  objects,  i.e.,  invariants  across  utterances.)  All  individuals,  including  spatio- 
temporal  locations,  have  properties  (like  being  fragile  or  red)  and  stand  in  relations  to  one  another  (like 
being  earlier,  being  under). 

The  earliest  formal  notion  of  situation  was  introduced  by  Barwise  in  [6]  as  a  means  of  giving  more 
realistic  formal  semantics  for  speech  acts  than  what  was  then  available.  In  contrast  with  a  "world"  which 
determines  the  value  of  every  proposition,  a  situation  corresponds  to  the  limited  parts  of  reality  we 
perceive,  reason  about,  and  live  in.  With  limited  information,  a  situation  can  provide  answers  to  some  but 
not  all  questions  induced  by  the  decision  problem. 

Situation  Semantics  is  a  semantic  framework  in  which  basic  properties,  relations,  events  and  even 
situations  are  reified  (i.e.,  made  concrete)  as  objects  to  be  reasoned  about  [7]:  "One  of  the  starting  points 
for  situation  semantics  was  the  promotion  of  real  situations  from  second  class  citizens  to  first  class 
citizens.  By  a  situation,  then,  we  mean  a  part  of  reality  that  can  be  comprehended  as  a  whole  in  its  own 
right  -  one  that  interacts  with  other  things.  By  interacting  with  other  things  we  mean  that  they  have 
properties  or  relate  to  other  things."  Note  that  once  a  situation  is  made  into  a  concrete  object,  various 
properties  can  be  associated  with  the  situation.  That  is  to  say  that  the  objects  may  be  complex, 
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incorporating  both  other  objects  as  well  as  their  properties  and  relations  among  the  objects.  It  can  even 
include  other  situations.  While  Barwise's  situation  semantics  is  only  one  of  the  many  alternative  semantic 
frameworks  currently  available,  its  basic  themes  have  been  incorporated  into  most  others. 

In  Situation  Semantics  invariants  and  uniformities  across  situations  are  highlighted,  allowing  the 
formation  of  particular  views  of  reality:  a  flow  of  types  of  situations  related  by  uniformities  like  e.g. 
individuals,  relations,  and  time-space-locations.  These  uniformities  constrain  a  system's  external  world  to 
become  its  view  of  reality  as  a  specific  fragment  of  persistent  (and  remembered)  courses  of  events  whose 
expectability  renders  them  interpretable. 

According  to  Situation  Semantics  [8],  any  language  expression  is  tied  to  reality  in  two  ways:  by  the 
discourse  situation  allowing  an  expression's  meaning  to  be  inteipreted  and  by  the  described  situation 
allowing  its  interpretation  to  be  evaluated  truth-functionally.  Our  aim  is  to  extend  this  assertion  from  any 
language  expression  to  any  high  level  piece  of  information. 

SAW  plays  a  central  role  in  cognition,  which  spans  the  entire  spectrum  of  cognitive  activities,  from 
situation  analysis  (perception  +  comprehension)  to  near  future  forecast.  All  of  these  processes  (perception, 
identification,  and  interpretation  of  external  structures)  are  considered  information  processing  which 
(natural  or  artificial)  systems  are  able  (or  unable)  to  perform.  Human  beings  appear  to  be  very  particular 
information  processing  systems  whose  outstanding  plasticity  and  capability  to  cope  with  changing 
situational  conditions  is  essentially  tied  to  their  use  of  natural  languages  in  communication  to  acquire 
knowledge.  Their  knowledge  based  processing  of  information  makes  them  cognitive,  and  their  sign  and 
symbol  generation,  manipulation,  and  understanding  capabilities  render  them  semiotic. 

Semiotics  is  the  general  theory  of  signs.  It  can  be  broadly  defined  as  a  domain  of  investigation  that 
explores  the  nature  and  function  of  signs  as  well  as  the  systems  and  processes  underlying  signification, 
expression,  representation,  and  communication.  "Semiotic  Cognitive  Information  Processing  (SCIP)  is 
inspired  by  information  systems  theory  according  to  which  living  systems  process  and  structure 
environmental  data  according  to  their  own  structuredness."  [8].  The  interest  of  SCIP  is  the  study  of 
cognitive  foundations  of  sign  organization  and  manipulation  processes.  The  main  domain  for  the 
application  of  semiotic  cognitive  information  processing  is  currently  restricted  to  natural  language. 

2.2.2  Deficiency  of  current  information  fusion  technology 

Information  Fusion  encompasses  the  theory,  techniques  and  tools  conceived  and  employed  for  exploiting 
the  synergy  in  the  information  acquired  from  multiple  sources  (sensors,  databases,  information  gathered 
by  humans,  etc.)  such  that  the  resulting  decision  making  or  action  is  in  some  sense  better  then 
(qualitatively  or  quantitatively,  in  terms  of  accuracy,  robustness,  etc.)  than  would  be  possible  if  any  of 
these  sources  were  used  individually  without  such  synergy  exploitation. 

The  field  of  information  fusion  is  a  multidisciplinary  research  area  and  overlaps  with  a  number  of  other 
areas  that  have  their  own,  partly  overlapping,  research  and  development  communities:  cognitive  sciences, 
mathematics  and  computer  sciences.  Over  the  years,  a  lot  of  fusion  models  have  been  proposed.  However, 
no  model  has  become  as  influential  in  data  fusion  as  the  Joint  Directors  of  Laboratories  (JDL)  or  the 
Waterfall  fusion  model.  However,  while  the  JDL  data  fusion  model  provides  a  methodology  for 
organization  of  research  in  fusion  [10],  it  doesn't  provide  an  architectural  framework  for  design  of 
systems,  or  model  it  from  a  cognitive  perspective. 

In  [1]  and  [11],  Endsley  provides  an  alternative  to  the  JDL  model  that  addresses  SAW  from  this  viewpoint 
(i.e.  Mental  Model).  This  model  is  defined  as  "the  perception  of  the  elements  in  the  environment  within  a 
volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the 
near  future". 
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Current  fusion  methods  are  built  around  the  hypothesis  of  a  "closed  world".  This  hypothesis  is  a  modelling 
simplification  but  it  does  not  reflect  the  real  world.  Future  command  support  systems  will  have  to  cope 
with  missions  with  low  background  intelligence  and  little  knowledge  about  the  behaviour  of  the  adversary. 
Therefore,  keeping  this  closed  world  hypothesis  imposes  on  designers  the  need  to  create  or  constantly 
modify  the  models  used  for  SAW  [12]. 

Information  fusion  has  also  been  studied  from  a  high-level  cognitive  point  of  view  in  the  logic-based 
artificial  intelligence  community.  The  focus  is  then  on  the  fusion  and  merging  of  several  sources  of 
knowledge,  taking  the  various  possible  epistemological  role  of  information  into  account  (knowledge, 
beliefs,  goals,  plans,  uncertain  data  etc.) 

Kokar,  in  [13],  argues  that  it  is  a  much  more  difficult  problem  to  automatically  construct  a  good  human- 
oriented  representation  of  a  situation  (SAW)  than  to  achieve  situation  assessment  as  it  is  described  in  the 
Level  2  of  the  JDL  data  fusion  model,  because,  SAW  is  not  only  knowing  objects  but  also  relations 
between  them.  Therefore,  the  four  main  difficulties  are  the  assessment  of  the  relations,  their  relevance,  the 
complexity  of  their  derivation  algorithms  and,  the  uncertainty  dimension  of  these  objects  and  their 
relations. 

In  spite  of  significant  progress  in  research  on  information  fusion,  there  is  still  a  deficiency  of  a  unified 
theoretical  framework  and  a  human-oriented  modelling  approach  for  information  fusion.  This  is  a  major 
impediment  to  the  development,  evaluation,  comparison  or  reuse  of  fusion  technology  and  systems  in 
CSS. 


2.3  Our  Vision 

We  claim  that  a  CSS  can  relax  the  cognitive  overload  and  construct  the  current  situation  in  a 
comprehensible  way,  based  on  a  situation  model  and  a  generalized  information  fusion  engine.  Our  vision 
relies  on  two  strategic  foundations: 

•  Model  user  perception:  dealing  with  cognitive  overload  requires  a  model  of  the  user's  cognitive 
situation  in  terms  of  a  situation  model  which  enables  the  semantic  interpretation  of  each  type  of 
information.  The  situation  model  can  be  seen  as  a  unified  semantic  universal  model  supporting  the 
understanding  process  of  types  of  information  with  distinct  initial  semantics.  The  key  idea  is  to 
generalize  the  notion  of  context  in  natural  language  understanding  to  all  other  types  of 
information. 

Based  on  the  Gestalt  theory,  our  aim  is  to  analyse  the  notions  of  perceptive  items,  grouping  and 
good  form  in  the  information  synergy  domain.  The  idea  is  to  apply  the  Gestalt  theory  to 
information  synergy  by  extending  or  redefining  the  laws  that  have  been  defined  for  visual 
perception  (and  particularly  proximity,  similarity  and  continuity  laws)  to  the  perception  of  any 
kind  of  information.  This  will  allow  us  to  build  a  picture  of  what  is  understood  by  the  individuals. 
Based  on  that  picture,  we  believe  that  efficient  information  synergy  construction  will  be  possible 
because  individual  needs  and  perception  will  be  taken  into  account. 

As  Situation  Semantics  argues  that  it  is  sufficient  and  fruitful  to  start  with  partial  description  or 
partial  information,  we  propose  to  adapt  the  models  developed  in  Situation  Semantics  approach  to 
our  purpose:  to  represent  a  reality  described  through  several  heterogeneous  pieces  of  information, 
stemming  from  diverse  sources  with  different  trust  and  precision  levels. 

We  will  also  study  the  extension  of  SCIP  from  natural  language  based  information  (texts,  speech) 
to  any  type  of  information. 


RTO-MP-IST-055 


3-5 


Cognitive  Situation  Awareness  for  Information  Superiority 


ORGANIZATION 


•  Revisit  information  fusion  methods  and  Generalise  information  fusion  mechanisms:  relevant 
information  fusion  will  be  revisited  along  two  dimensions:  the  systematic  account  of  situation 
model  as  provided  above,  and  a  systematic  investigation  (existing  gaps  in  the  efficiency  of  the 
method  and  identification  of  solutions)  of  the  relevance  for  other  types  of  information.  Finally,  the 
revisited  fusion  methods  will  be  merged  into  a  single  theoretical  framework.  This  framework  will 
formalize  all  the  previously  identified  fusion  methods  and  will  be  extended  to  all  other  types  of 
information. 

An  information  fusion  engine  will  carefully  combine  the  relevant  fusion  approaches  for  several 
information  sources  (e.g.  text  and  audio,  text  and  image...)  in  order  to  construct  a  comprehensible 
situation  for  the  user. 

Our  aim  is  to  render  a  relevant  but  comprehensive  situation  awareness  picture  supported  by  cognitive 
information  fusion.  To  address  that  goal,  the  very  substance  is  deeply  rooted  in  "Situation  Semantic 
theory",  in  SCIP  and  in  "fusion".  Before  being  able  to  fuse  information  coming  from  heterogeneous 
sources,  it  is  necessary  to  define  a  unified  information  frame. 

Some  challenges  of  Information  Fusion  for  CSS  are  to  transform  the  raw,  imprecise,  conflicting  and  often 
paradoxical  information  received  from  the  different  sources  into  statements  understandable  by  both  man 
and  machines,  the  ability  to  adapt  efficiently  to  changing  environmental  conditions,  learning  how  to 
anticipate  possible  changes  in  its  environment  is  inescapable.  Combining  the  semiotic  with  the  cognitive 
paradigm,  will  allow  us  to  define  information  perception  and  comprehension  models  for  a  SAW  system  in 
which  the  user  will  be  able  to  browse  without  cognitive  overload. 

Semiotic  cognitive  information  processing  where  dispersed  information  coming  from  discrete  sources  are 
analysed  through  cognitive  prisms  and  correlated  to  provide  the  right  information  at  the  right  time  should 
support  the  system.  The  goal  is,  thus,  to  establish  a  process  of  multi-source  information  fusion,  where  a 
qualitatively  new  meaning  is  assigned  to  the  fused  data. 


3.0  PROPOSED  APPROACH 

3.1  Terminology 

For  a  matter  of  comprehension,  we  define  here  several  of  the  terms  that  we  will  use  in  the  following 
sections  of  this  paper. 

•  Domain:  The  domain  is  the  operational  context,  which  defines  the  point  of  view  to  apply  on  the 
raw  data.  It  gives  the  significance  of  the  activity.  The  domain  knowledge  is  the  amount  of 
information  that  is  known  by  the  operator. 

•  Situation:  The  situation  is  the  whole  set  of  events  and  their  actors  that  are  happening  on  a  field  of 
observation.  The  situation  can  be  expected,  that  is  to  say  that  it  hasn’t  been  observed  yet,  but  a  set 
of  events  are  expected  to  happen  on  the  field  of  observation.  In  this  case,  we  call  this  situation 
sought-after  situation  or  abstract  situation.  The  abstract  situation  is  somehow  the  formal 
description  of  specific  domain  knowledge.  On  the  contrary,  a  situation  can  be  observed. 

•  Mission:  The  mission  of  an  operator  is  the  task  that  has  been  assigned  to  him.  In  our  domain,  it  is 
comprised  the  observation  of  the  field  of  action,  making  deductions  and  projections  into  the 
future,  applying  strategy  rules  and  to  making  decisions. 

•  Observation:  An  observation  is  a  small  part  of  a  situation  that  has  been  “seen”.  In  our  case,  an 
observation  is  the  result  of  the  analysis  of  an  input  document,  such  as  a  description  of  entities  on 
the  field  of  action,  a  radio  recording  etc.  Several  different  observations  can  correspond  to  a  single 
event  or  entity  as  observations  are  subjective. 
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3.2  Hypothesis 

Our  approach  concentrates  on  document  analysis  for  situation  comprehension  and,  more  particularly,  on 
the  fusion  process  of  information  extracted  from  several  documents  in  order  to  build  a  clear  vision  of  an 
ongoing  situation.  Therefore,  we  suppose  that  documents  have  been  filtered  in  a  previous  step.  This 
filtering  step  is  supposed  error  free,  so  that  all  documents  arriving  to  our  system  are  relevant  to  the 
operator’s  mission  and  have  to  be  analysed  in  detail. 

Because  of  its  genericity,  our  approach  will  allow  the  inclusion  of  any  type  of  document  in  the  analysis 
process,  but  in  our  first  study,  we  will  concentrate  on  textual  news  or  image  description  and  phone  or  radio 
tapping.  These  restrictions  are  due  to  the  availability  of  information  extraction  systems.  The  information 
extraction  systems  that  we  work  with  are  able  to  process  textual  and  audio  documents  only. 

3.3  Information  flow  and  situation  analysis 

In  this  section,  we  briefly  present  the  flow  of  information  in  our  system  as  well  as  the  different  steps  of  the 
analysis.  Figure  1  summaries  this  information  flow  as  well  as  the  processes  applied  on  it. 


Situation 

Representation 


Figure  1:  Information  Flow 


Our  approach  relies  on  two  separate  blocks  of  processes:  the  Situation  Representation  and  the  Situation 
Analysis.  The  inputs  for  these  processes  are  the  operator’s  mission,  the  domain  knowledge  and  the 
incoming  row  data  stemming  from  the  available  captors.  We  choose  to  separate  the  two  blocks  of 
processes  because  of  the  difference  in  their  nature  (one  processes  static  information,  while  the  other  one 
takes  dynamic  aspects  into  account)  and  their  complementarities. 

In  the  Situation  Representation  part,  the  ongoing  situation  is  understood  at  any  moment  through  the 
analysis  of  incoming  information.  A  picture  of  the  ongoing  situation  is  built  which  doesn’t  take  into 
account  the  dynamic  vision  of  the  situation  nor  is  any  inference  made  to  plan  a  possible  future  state  of  the 
situation.  To  build  this  picture,  relevant  information  is  extracted  from  the  input  data,  on  the  basis  of  what 
has  been  defined  as  relevant  regarding  the  static  elements  of  the  operator’s  mission  and  the  domain 
knowledge.  The  relevant  information  is  expressed  through  entities  representing  actors  and  events  and 
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relations  among  them.  As  it  is  not  the  main  object  of  our  paper,  the  extraction  process  won’t  be  detailed  in 
the  following  sections.  We  will  only  focus  on  the  structure  that  we  have  chosen  to  represent  domain 
knowledge  and  the  sought-after  situation. 

The  second  process  occurring  for  Situation  Representation  is  a  reification  and  fusion  process.  The 
extracted  information  is  aggregated  into  a  common  structure,  representing  the  ongoing  situation.  The 
aggregation  is  realised  based  on  a  reification  of  the  Abstract  Situation  thanks  to  observations  (i.e.  entities 
and  relations)  extracted  from  incoming  data.  That  fusion  could  be  needed  between  partial  observations  in 
order  to  instantiate  the  abstract  situation.  The  result  of  this  process  is  the  representation  of  the  ongoing 
situation  (Concrete  Situation),  given  the  observations  available  through  the  input  document  analysis.  The 
structure  used  in  this  step  is  based  upon  the  one  used  to  describe  the  expected  situation  (Abstract 
Situation)  and  is  a  semantic  graph,  partially  instantiated. 

The  Situation  Analysis  part  focuses  on  understanding  the  dynamic  of  the  ongoing  situation.  This  analysing 
part  is  based  on  the  study  of  the  dynamic  aspects  of  the  operator’s  mission,  regarding  the  different  pictures 
of  the  situation  build  by  the  Situation  Representation  part  and  their  temporal  relations. 

Once  the  situation  is  represented,  reasoning  upon  it  can  take  place.  We  use  the  Complex  Event  Processing 
(CEP)  approach  to  realize  this  step  of  the  analysis.  Based  on  the  representation  of  the  situation  of  the 
previous  step,  events  corresponding  to  the  current  state  of  the  situation  are  generated  and  sent  to  the 
reasoning  module.  During  the  reasoning  step,  changes  to  the  graph  representing  the  situation  can  be 
proposed  to  the  operator.  Indeed,  the  reasoning  engine  can  detect  changes  in  the  situation  (due  to  temporal 
aspects  for  instance)  or  unlinked  elements  that  belong  to  the  same  event  (proposition  of  fusion  of  events). 
It  can  even  create  new  knowledge  based  on  business  reasoning  rules  for  instance. 


4.0  SITUATION  REPRESENTATION 

4.1  From  raw  data  to  entities 

Our  work  will  concentrate  on  the  representation  of  a  situation  and  its  analysis.  In  order  to  execute  these 
two  processes,  a  first  step  is  necessary:  the  extraction  of  entities  and  relations  among  entities  from  the 
input  information  sources.  The  entities  and  relations  that  are  to  be  extracted  from  the  source  documents  are 
actors  and  events  of  the  observed  situation  as  well  as  the  relations  amongst  them. 

This  step  of  identification  and  extraction  of  actors  and  events  can  be  either  done  manually,  or  semi- 
automatically  with  systems  such  as  Sem+  [14]  for  entities  and  relations  extraction  from  textual  coipus. 
Sem+  allows  the  operators  to  express  relations  they  are  looking  for  through  examples  of  utterances 
expressing  the  sought-after  relations.  The  examples  are  then  processed  by  the  system  in  order  to  define 
patterns  of  the  required  relations.  The  patterns  are  then  used  on  the  input  corpus  and  relations  are 
extracted.  Our  work  focuses  on  the  processes  that  follow  the  extraction  step. 

4.2  Abstract  description  of  the  situation 

Gestaltist  principles  of  perception  rely  on  the  idea  that  context  is  very  important  in  perception  and  thus  in 
the  way  our  mind  interprets  observations  to  elaborate  a  picture  of  the  whole  underlying  situation.  What 
makes  someone  able  to  build  complete  and  complex  situations,  out  of  a  set  of  partial  observations  of  these 
situations,  is  the  fact  that  the  complete  situations  are  somehow  familiar.  Therefore,  typical  patterns,  such 
as  known  relations  between  entities,  can  be  observed  and  recognized  and  from  then  on  a  human  mind 
creates  representations  of  the  missing  elements  of  the  situation,  even  if  they  can’t  be  observed,  in  order  to 
build  an  internal  picture  of  the  whole  situation  and  reason  about  it.  This  process  explains  how  decisions 
can  be  taken  even  in  situations  where  all  the  single  elements  are  not  known,  provided  that  the  operators  are 
well  experienced  and  thus  know  exactly  what  they  are  looking  for,  when  observing  a  field  of  action. 
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As  we  want  to  provide  cognitive  functions  to  CSS,  we  propose  to  follow  and  apply  Gestalt  principles  to 
situation  perception.  The  observed  situation  has  therefore  to  become  somehow  familiar  to  the  CSS. 
Therefore,  we  propose  to  explicitly  define  the  typical  features  of  the  business  domain  and  of  the  sought- 
after  situation.  Whenever  these  features,  or  a  subset  of  them,  will  be  observed  (i.e.  extracted  from  input 
documents),  we  will  suppose  that  the  situation  has  been  found  (with  a  degree  of  certainty  that  will  depend 
on  observations,  their  reliability  and  the  proportion  of  observed  elements  among  all  the  sought-after 
elements).  Inferences  will  then  be  allowed  according  to  the  possible  evolutions  in  time  and  space  of  the 
situation,  and  thus  help  will  be  provided  to  the  operator  in  order  to  make  decisions. 

On  the  contrary,  if  incoherent  elements  related  to  a  situation  that  was  supposed  to  be  recognised  are 
detected  (observed),  automatic  analysis  of  them  will  emphasize  the  inconsistencies  and  will  then  thwart 
cognitive  and  human  analysis  of  the  situation,  allowing  operators  to  review  the  previous  analysis  and 
inferences  in  order  to  detect  a  change  in  the  situation  as  soon  as  possible.  We  believe  that  automatic  and 
mechanic  analysis  of  the  situation  has  to  be  closely  linked  to  human  and  cognitive  reasoning  in  order  to 
avoid  errors  in  perception  and  comprehension  levels  by  rising  alerts  when  both  analysis  don’t  match 
anymore.  Cognitive  capabilities  of  humans  have  to  be  supported  by  large  set  of  data  processing 
capabilities  of  automatic  information  systems.  There  must  be  collaboration  between  the  two,  hence 
cognitive  functionalities  have  to  be  integrated  in  the  CSS. 

To  describe  the  domain  knowledge,  or  as  we  called  it  before,  the  sought-after  situation,  we  choose  to  use  a 
conceptual  graph.  The  nodes  of  the  graph  will  describe  the  actors  and  events  of  the  situation  and  relations 
that  can  exist  between  them  will  link  them  to  each  other.  The  sought-after  situation  will  be  an  abstract 
graph:  the  nodes  of  the  graph  won’t  be  concrete  instances  of  actors  and  events  but  will  be  labelled  with 
their  conceptual  types.  The  relations,  as  for  them,  are  conceptual  relations  such  as  composition,  part-of, 
etc. 


4.3  Instantiation  of  the  situation 

Once  the  abstract  situation  is  defined,  observations  made  through  the  analysis  of  input  documents  will  be 
used  to  build  a  (partially)  concrete  one.  This  concrete  situation  is  mainly  the  observed  situation  but 
inferred  elements  can  also  be  included,  as  we  will  see  in  the  following  section.  The  first  step  to  be  realised 
in  order  to  instantiate  the  situation  is  to  collect  all  the  entities  and  relations  extracted  from  the  analysed 
input  document  and  identify  the  nodes  to  which  it  must  be  linked.  The  result  of  this  operation  is  a 
collection  of  instantiated  pieces  of  graph,  representing  a  collection  of  observations  of  the  situation. 

Once  the  collection  of  pieces  of  graph  are  available  they  will  be  analysed,  regarding  on  the  one  hand,  the 
abstract  situation  and,  on  the  other  hand,  the  distance  calculated  between  the  different  nodes,  in  order  to 
process  to  a  fusion  step.  The  observations  (pieces  of  nodes)  and  the  abstract  situation  will  be  gathered 
together  so  to  build  a  whole  (partially  un-instantiated)  picture  of  the  complete  situation.  The  different  parts 
of  the  graph  are  weighted  according  to  the  trust  accorded  to  the  different  sources  of  information  that 
provided  the  related  observation. 

The  construction  of  the  pieces  of  graph  representing  the  observations,  as  well  as  the  fusion  of  two  pieces 
of  graph  into  a  single  one  can  require  mechanisms  to  solve  reference  problem.  These  reference  problems 
can  occur  while  analysing  a  single  document  (e.g.  several  paragraphs  of  one  text)  but  also  between  several 
documents  dealing  with  a  single  object.  We  will  use  cognitive  mechanisms  to  solve  this  sort  of  problem. 
The  salience  of  the  observed  facts,  for  instance,  will  allow  us  to  solve  some  of  the  reference  problems. 
More  generally,  reference  resolution  methods  exist  for  dialog  analysis,  which  are  based  upon  natural 
language  analysis.  We  plan  to  extend  and  modify  them  if  needed  to  resolve  reference  between  several 
sources  of  information. 
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5.0  SITUATION  ANALYSIS 

Whereas  the  previous  sections  presented  our  approach  to  static  situation  representation  (i.e.  a  picture  of  the 
situation  at  a  given  moment),  our  aim  is  to  allow  our  system  to  reason  about  a  dynamic  situation  given  a 
specific  mission,  in  order  to  make  decision.  Therefore,  subjectivity  and  mission  specificities  have  to  be 
taken  into  account.  Moreover,  the  mission  of  an  operator  is  specific  and  expressed  through  business  rules 
(for  instance  strategy  rules,  deductions  that  can  be  made  given  a  set  of  observations  and  temporal 
development...).  It  is  very  different,  in  structure  and  in  meaning  from  general  knowledge  of  the  domain 
that  is  used  to  express  the  sought-after  situation.  The  use  of  the  same  graph  structure  as  the  one  used  to 
build  the  picture  of  the  situation  for  representing  this  reasoning  knowledge  is  thus,  in  our  opinion  not 
appropriate.  We  propose  to  use  a  reasoning  module  based  on  mission-specific  rules  and  events  observed 
and  thus  induced  by  the  situation  representation  and  are  specific  to  the  operator’s  mission. 

5.1  Analysing  the  situation 

To  manage  situation  analysis,  we  propose  to  follow  the  Complex  Event  Processing  (CEP)  approach  [15]. 

CEP  is  a  set  of  techniques  and  tools  to  help  operators  understand  and  control  event  driven  information 
systems.  The  goal  of  CEP  is  to  enable  the  information  contained  in  the  events  flowing  through  an 
information  system  to  be  discovered,  understood  in  terms  of  its  impact  on  high-level  management  goals 
and  business  processes  and  acted  upon  in  real  time.  CEP  employs  techniques  such  as  detection  of  complex 
patterns  of  many  events,  events  correlation  and  abstraction,  event  hierarchies,  and  relationships  between 
events  such  as  causality,  membership  and  timing  ordering. 

As  we  derive  an  abstract  situation  graph,  we  define  some  mission  and  knowledge  patterns,  which  set 
alarms  on  when  events  situation  match  them.  It  is  as  an  inference  rules  base  engine  but  not  a  classical  one. 
It  takes  into  account  the  dynamic  aspect  of  a  situation  (through  the  continuous  flow  of  events)  and  the  life 
time  duration  of  an  event. 

The  analysis  of  the  ongoing  situation  will  allow  the  system  to  automatically  raise  alerts  when  some 
inconsistency  is  detected,  or  when  a  strategic  business  rule  has  to  be  applied.  The  picture  of  the  situation 
(situation  representation)  will  also  be  enriched  according  to  deduction  made  by  the  reasoning  module.  In 
this  case,  if  knowledge  is  added  in  the  situation  representation,  a  typical  level  of  confidence  will  be  used  in 
order  to  quote  this  new  knowledge  as  being  inferred  and  not  observed. 

5.2  Typology  of  events  for  situation  analysis 
Mission  and  knowledge  patterns 

The  mission  patterns  are  the  events  aggregation  to  match  describing  the  mission  of  the  operator,  as  well  as 
the  state  of  accomplishment  of  this  mission.  These  patterns  are  for  instance  business  rules  that  the  operator 
applies  in  order  to  make  decisions  or  infer  the  future  state  of  the  situation.  Mission  patterns  can  be  strategy 
rules  for  instance.  The  knowledge  patterns  are  the  same  but  derived  from  the  domain  knowledge.  Some 
patterns  can  be  the  result  of  both  mission  and  domain  knowledge. 

The  situation  analysis  consists  in  matching  these  patterns  with  situation  and  control  events,  and  then 
provides  decision  events  (which  can  be  re-injected  in  the  module  as  new  situation  events). 

Situation  events 

Situation  events  concern  the  situation,  taking  into  account  the  operator’s  mission  and  giving  value  of  some 
of  the  observations  made.  They  are  useful  pieces  of  information  for  the  decision  making  process  of  the 
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operator  and  don’t  necessary  reflect  the  whole  observed  situation.  These  events  are  generated  by  the 
situation  representation  module  and  sent  to  the  situation  analysis  module  each  time  a  change  is  detected  in 
the  graph  representing  the  current  state  of  the  ongoing  situation.  They  are  dated  in  order  to  allow 
reasoning  according  to  temporal  development  of  situation  changes.  Each  time  nodes  in  the  situation  graph 
will  be  fused  and  each  time  a  new  observation  will  be  introduced  in  the  complete  situation  representation, 
replacing  an  abstract  concept,  one  or  more  of  these  events  will  be  generated  in  order  to  make  any  change 
in  the  current  situation  explicit. 

Note  that,  following  the  Gestalt  principles,  events  can  be  generated  even  if  no  observation  has  been  made 
in  order  to  simulate  the  cognitive  way  of  building  whole  out  of  incomplete  pieces.  A  particularly  low 
weight  will  then  be  associated  with  these  events  as  they  are  particularly  un-trustful  because  being  un¬ 
observed. 

Control  events 

Control  events  are  process  management  events.  They  result  from  pattern  matching  rules  but  are  not 
enough  to  conclude  on  the  situation  analysis.  They  are  used,  for  instance,  to  manage  the  correlation  of 
several  events. 

Decision  events 

Decision  events  are  events  result  from  the  situation  analysis  process  and  give  clues  concerning  the 
decision  that  has  to  be  made  by  the  operator.  Decision  events  are  produced  when  the  business  rules 
defining  the  mission  of  the  operator  are  applied  to  both  situation  and  mission  events.Decision  events  can 
lead  to  the  proposition  of  the  fusion  of  several  nodes  in  the  situation  representation  graph.  The  operator 
will  supervise  any  fusion  in  the  situation  representation. 

New  knowledge  resulting  from  decision  events  can  also  be  created.  Then,  new  instantiated  graph  pieces 
will  be  introduced  in  the  situation  representation  module.  These  pieces  of  graph  will  have  a  particular 
(low)  weight,  indicating  that  the  knowledge  introduced  was  not  observed  directly  but  derived  from 
business  rules.  This  creation  of  new  knowledge  corresponds  to  a  projection  of  the  evolution  of  the 
observed  situation  into  a  future  state. 


6.0  CONCLUSION  AND  FUTURE  WORKS 

We  proposed  a  general  approach  to  information  analysis  for  situation  representation  and  analysis.  This 
approach  is  human  centred  as  well  as  cognitive  because  a  human  is  supervising  the  whole  process  and  the 
system  is  aimed  at  helping  the  operators  in  their  mission  by  supplying  useful  information.  We  claim  that 
integrating  cognitive  functionalities  into  CSS  will  provide  useful  assistance  to  the  operators. 

However,  we  now  have  to  define  the  different  algorithms  that  are  necessary  to  realise  our  system.  This 
includes  the  calculation  of  distance  between  two  nodes  in  the  situation  graph,  the  definition  of  fusion 
operators  between  nodes  in  the  graph  and,  more  generally,  the  definition  of  the  properties  that  are 
necessary  for  the  abstract  and  concrete  situation  graphs.  These  processes  and  properties  must  take  into 
account  the  specificities  of  the  domain  in  which  we  are  working:  two  nodes  representing  two  observations 
of  the  same  object  can  each  be  related  to  several  other  nodes.  The  two  sets  of  related  nodes  can  be 
different  or  partially  equals.  In  the  same  way,  two  nodes  representing  the  same  physical  object  can  have 
different  attributes  because  the  observations  from  which  they  result  take  different  points  of  view  into 
account  or  stem  from  different  sources. 

Concerning  the  cognitive  functionalities,  one  of  the  possible  extensions  that  we  would  like  to  study  is  how 
to  apply  the  gestaltist  notions  to  different  types  of  information.  Indeed,  these  notions  have  been  widely 
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studied  in  field  such  as  Human-computer  interaction,  but  we  would  like  to  generalize  the  approach  to 
situation  perception  in  general.  The  aim  is  to  highlight  the  elements  that  make  a  situation  to  look  familiar 
as  well  as  the  characteristics  that  drive  humans  to  recognize  a  whole  situation  out  of  incomplete  and  local 
observations  and  descriptions.  On  the  other  hand,  we  would  like  to  extract  typical  features  that  allow 
human  operators  to  realize  they  made  mistakes  on  a  situation  recognition  and  evaluation. 
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Motivation  O 


i 


Information  Superiority  is  the  capability  to  collect, 
process  and  disseminate  an  uninterrupted  flow  of 
information  while  exploiting  or  denying  an  adversary’s 
ability  to  do  the  same. 


■  Large  amount  of  data  &  different  information  sources 

■  more  global  view  of  the  situation 

■  but  cognitive  overload:  errors  in  perception,  comprehension, 
projection 


■  Efficient  Command  Support  System  necessitates 

■  Clear  vision  of  the  situation 

■  Cognitive  functions  to  support  decision  making 
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Presentation  overview  O 


i 


■  Limitations  of  current  command  support  systems 

■  Our  vision 

■  Proposed  approach 

■  The  use  of  knowledge  on  raw  information 

■  Situation  recognition 

■  From  local  sub-situations  to  the  global  situation 

■  Situation  analysis 

■  Conclusion 
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Limitation  of  current  command  support  systems  O 


i 


The  vital  pieces  of  information  are  spread  across  several 

formats  and  must  be  combined  to  get  the  situation  desired 

at  the  right  time. 

Up  to  now: 

■  Each  format  is  processed  on  the  basis  of  distinct  theories 

■  No  sound  framework  for  situation  understanding  and 
integration  of  disparate  sources  of  information  is  available 

■  No  model  is  available  for  supporting  in-depth  understanding 
of  the  situation 

■  JDL  doesn't  provide  an  architectural  framework 

■  Hypothesis  of  a  “closed  world”:  possible  alternatives  are 
only  those  known 
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Our  vision  ® 

Give  more  time  for  the  decision  making  phase  by: 

■  minimising  low  level  tasks 

■  rendering  a  synthetic  view  of  the  current  situation 

■  processing  its  possible  evolutions 

We  propose  to: 

■  automate  systematic  (non  decisional)  processes  (in  a  supervised 
approach) 

■  enhance  the  user’s  capabilities  of  perceiving  and  analysing  the 
situation  and  its  evolution 

■  model  user  perception  in  order  to  prevent  errors  in  perception, 
comprehension  or  projection  level 

This  will: 

■  give  an  advance  to  the  user  over  other  actors 

■  make  the  disinformation  actions  more  difficult  thanks  to  the 
manipulation  of  a  more  diversified  and  important  information  flow 
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Proposed  approach  ® 

global  framework  which  uses  raw  information  to  give  a  synthesis  of  a 
situation  by  the  application  of  a  domain  knowledge  point  of  view. 
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Example  :  road  traffic  surveillance  O 


Domain  knowledge:  Supervision  of  the  flow  of  vehicles  on  a  motorway 
Mission:  Traffic  jam  detection 


■  Road  characterized  by: 

■  number  of  lines 

■  number  of  directions/ways 

■  capacity  of  the  road  (number 
of  vehicles  /  hour) 

■  referring  speed 

■  Traffic  Jam  detected  by: 

■  slow  flow  of  vehicles 

■  high  density  of  vehicles 

■  in  the  same  location 


■  Potential  causes  of  traffic  jam: 

■  accident 

■  roadworks 

■  (unexplained)  slowing  down 

■  Raw  information  coming  from: 

■  video  surveillance  (computes 
traffic  speed  and  density) 

■  drivers’  phone  calls 

■  reports  from  road  safety 
services 
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Situation  Analysis 


■  From  knowledge 

■  Modelling  abstract  possible  situations 

•  graph  representation 

•  logical  analysis 

■  With  raw  information 

■  Text,  audio,  video,  image... 

■  Relevant  to  the  mission 

■  Including  trust  skill  for  sources 

■  To  Entities  and  relations  between 
them 

■  Extraction  techniques  parameterised 
by  the  abstract  situations 

■  Trust  on  sources  becomes  belief  tor 
the  resulting  information 
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Example  on  Situation  Representation  O 


T 


Road  Traffic 
Traffic  flow 

-  speed 

-  density 

-  location 

Perturbation 

-  location 


Perturbation 


unplanned 


planned 


accident  undetermined  roadworks 


Abstract  Situation 
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Example  on  Situation  Analysis  O 


T 


■  Rules  derived  from  mission  and  domain  knowledge: 


■  If  the  flow  of  vehicles  is  slow  and  the  density  is  high  in  the 
same  location,  then  a  traffic  jam  is  detected 


■  If  there  is  an  accident  on  one  lane  of  a  2-way  road,  then  there 
is  a  high  probability  that  the  flow  of  vehicles  will  slow  down  on 
the  other  way. 


■  If  there  is  an  accident  on  one  lane  then  the  flow  will  slow 
down  of  X%  and  the  density  will  grow  up  of  Y%. 
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Situation  recognition  O 


T 


■  Use  of  similarity  operators  on  extracted 
information,  independently  of  their 
uncertainty,  to  group  redundant 
information  (graph  theory,  data  fusion 
approach,  natural  language  processing) 

■  Elimination  of  redundancy  with 
uncertainty  decision  aid  operators 
(possibilities  theory) 

■  Knowledge  emergence  by  combining 
the  abstract  situation  with  extracted 
information  to  obtain  concrete  sub¬ 
situations  (information  fusion) 


All  actions  are  supervised  by  the 
user  in  an  active  and/or  a  passive 
way 
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From  local  sub-situations  to  the  global  situation  O 


T 


■  Sub-situations  are  set  out  of  the 
dynamic  evolution  of  the  global 
situation 


■  The  conversion  of  a  sub-situation  is 
more  than  translation: 

■  filtering  of  sub-situations  according  to 
predefined  mission  and  knowledge 
patterns 

■  translation  of  filtered  entities  into 
events 
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Situation  analysis  O 
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Complex  Events  Pnxessing 
Pattern  matching 
Rules,  Agents,  Filters, 
Maps,  Constraints 


Use  of  Complex  Events  Processing 

technology  for  Situation  analysis. 

■  Mission  and  knowledge  patterns  are 

logical  and/or  causal  combination  of 

events 

■  Deductions  obtained  by  constrained 
and/or  conditional  inference  rules  where 
premises  are  those  patterns 


Decision  Events 

i ” _ cL 


Control  Events 


Situation  Events 


Situatlon  Analysis 


■  Provide  the  possibility  of  alerting  before  a 
state  of  a  situation  is  reached:  anticipation 
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Conclusion  O 


T 


It  is  much  more  difficult  to  automatically  construct  Situation  Awareness 
than  to  achieve  situation  assessment  as  it  is  described  in  JDL:  Situation 
Awareness  is  not  only  knowing  objects  but  also  relations  between  them. 

>  Challenges: 

■  Assessment  of  the  relations,  complexity  of  the  derivation 
algorithm, 

•  Derivation  of  a  priori  Knowledge  (Situation  Recognition  and  Situation 
Analysis)  semi-manually 

■  Relevance  of  the  relations, 

•  Situation  Recognition  :  computed  extraction  results,  fusion  and  trust  for 
sources 

•  Situation  Analysis  :  pattern  matching  and  belief  on  the  sub-situations 

■  Uncertainty  dimension  of  objects  and  relations 

•  Subjective  and  objective  human  perception  modelling 
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